Image processing device, image processing method, and program

ABSTRACT

There is provided an image processing device including: a data storage unit storing feature data indicating a feature of appearance of one or more physical objects; an environment map building unit for building an environment map based on an input image obtained by imaging a real space and the feature data, the environment map representing a position of a physical object present in the real space; a control unit for acquiring procedure data for a set of procedures of operation to be performed in the real space, the procedure data defining a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed; and a superimposing unit for generating an output image by superimposing the direction for each procedure at a position in the input image determined based on the environment map and the position information, using the procedure data.

CROSS-REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/069,389 (filed on Oct. 13, 2020), which is a continuation of U.S.patent application Ser. No. 16/668,862 (filed on Oct. 30, 2019 andissued as U.S. Pat. No. 10,810,803 on Oct. 20, 2020), which is acontinuation of U.S. patent application Ser. No. 16/263,524 (filed onJan. 31, 2019 and issued as U.S. Pat. No. 10,515,488 on Dec. 24, 2019),which is a continuation of U.S. patent application Ser. No. 16/026,763(filed on Jul. 3, 2018 and issued as U.S. Pat. No. 10,223,837 on Mar. 5,2019), which is a continuation of U.S. patent application Ser. No.15/713,801 (filed on Sep. 25, 2017 and issued as U.S. Pat. No.10,037,628 on Jul. 31, 2018), which is a continuation of U.S. patentapplication Ser. No. 14/991,477 (filed on Jan. 8, 2016 and issued asU.S. Pat. No. 9,805,513 on Oct. 31, 2017), which is a continuation ofU.S. patent application Ser. No. 14/869,063 (filed on Sep. 29, 2015 andissued as U.S. Pat. No. 9,754,418 on Sep. 5, 2017), which is acontinuation of U.S. patent application Ser. No. 14/527,148 (filed onOct. 29, 2014 and issued as U.S. Pat. No. 9,183,678 on Nov. 10, 2015),which is a continuation of U.S. patent application Ser. No. 12/984,847(filed on Jan. 5, 2011 and issued as U.S. Pat. No. 8,896,628 on Nov. 25,2014), which claims priority to Japanese Patent Application No.2010-021368 (filed on Feb. 2, 2010), which are all hereby incorporatedby reference in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing device, an imageprocessing method, and a program.

Description of the Related Art

Recently, technology called augmented reality (AR) in which an imageobtained by imaging a real space is processed and then presented to auser has been receiving attention. In the AR technology, usefulinformation related to a physical object in a real space present in aninput image may be inserted into the image and output as an outputimage, for example. That is, in the AR technology, typically, a largepart of the image presented to the user shows the real space, and a partof the image may be processed in accordance with a purpose. Such acharacteristic contrasts it with virtual reality in which an entire (ora large part) of the output image is composed using computer graphics(CG). By using the AR technology, for example, advantages such as easyrecognition of a situation of the real space by a user or operationsupport based on the output image may be provided.

In the AR technology, in order to present actually useful information tothe user, it is important that a computer accurately recognize asituation of the real space. Therefore, technology aimed at recognizingthe situation of the real space, which serves as a basis of the ARtechnology, has been developed. For example, Japanese Patent ApplicationLaid-Open Publication No. 2008-304268 discloses a method of dynamicallygenerating an environment map representing a three-dimensional positionof physical objects existing in a real space by applying technologycalled simultaneous localization and mapping (SLAM) capable ofsimultaneously estimating a position and a posture of a camera and aposition of a feature point shown in an image of the camera. Further, abasic principle of the SLAM technology using a monocular camera isdisclosed in Andrew J. Davison's, “Real-Time Simultaneous Localizationand Mapping with a Single Camera,” Proceedings of the 9th IEEEInternational Conference on Computer Vision Volume 2, 2003, pp.1403-1410.

Now, information terminals capable of performing enhanced informationprocessing are widely used and users use the information terminals toview various information. An example of the information includesdirections. For example, various information, such as table manners, acooking procedure, a method of manipulating or repairing electricalappliances, or the like, as well as directions of electronic devices,describing a procedure of an operation to be performed in a real spaceare made in an electronic form and provided to the users via web pagesor other applications.

SUMMARY OF THE INVENTION

However, existing electronic directions are generally described based onthe premise that the electronic directions are displayed on atwo-dimensional screen, like paper directions. Accordingly, someinformation about positions of operation in a three-dimensional realspace is missed, often making it difficult for a user to intuitivelyunderstand a procedure of the operation. In particular, for an operationin a real space in which an environment surrounding a user may bedynamically changed, such environment is not described in thedirections, making it more difficult to understand the directions.

Accordingly, if an environment map three-dimensionally representing anenvironment surrounding a user can be dynamically built and a directionfor each procedure of the operation can be displayed in a positionassociated with the environment map, the operation procedure is expectedto be intuitively and easily understood by the user.

In light of the foregoing, it is desirable to provide an imageprocessing device, an image processing method, and a program which allowan operation procedure performed in a real space to be intuitively andeasily understood by applying an environment map.

According to an embodiment of the present invention, there is providedan image processing device including: a data storage unit having featuredata stored therein, the feature data indicating a feature of appearanceof one or more physical objects; an environment map building unit forbuilding an environment map based on an input image obtained by imaginga real space using an imaging device and the feature data stored in thedata storage unit, the environment map representing a position of aphysical object present in the real space; a control unit for acquiringprocedure data for a set of procedures of operation to be performed inthe real space, the procedure data defining a correspondence between adirection for each procedure and position information designating aposition at which the direction is to be displayed; and a superimposingunit for generating an output image by superimposing the direction foreach procedure included in the set of procedures at a position in theinput image determined based on the environment map and the positioninformation, using the procedure data acquired by the control unit.

According to such a configuration, the direction for each procedureincluded in a set of procedures of operation to be performed in the realspace is superimposed at a position in the input image determined basedon the environment map three-dimensionally representing positions ofphysical objects present in the real space, according to definition byprocedure data.

The position information may designate a position in the environment mapat which the direction is to be displayed, by specifying a physicalobject related to the direction for each procedure.

The procedure data may define a further correspondence between thedirection for each procedure and a condition for progressing displayingof each direction.

The condition for progressing displaying of each direction may include acondition according to a position or a posture of a physical objectrepresented by the environment map.

The control unit may control displaying of the direction for eachprocedure included in the set of procedures according to the proceduredata.

The image processing device may further include a detection unit fordynamically detecting a position in the real space of the imaging devicebased on the input image and the feature data, and the position in theinput image at which the direction for each procedure is to besuperimposed may be determined according to the position in the realspace of the imaging device detected by the detection unit.

According to another embodiment of the present invention, there isprovided an image processing method in an image processing deviceincluding a storage medium having feature data stored therein, thefeature data indicating a feature of appearance of one or more physicalobjects, the method including the steps of: acquiring an input imageobtained by imaging a real space using an imaging device; building anenvironment map based on the input image and the feature data, theenvironment map representing a position of a physical object present inthe real space; acquiring procedure data for a set of procedures ofoperation to be performed in the real space, the procedure data defininga correspondence between a direction for each procedure and positioninformation designating a position at which the direction is to bedisplayed; and generating an output image by superimposing the directionfor each procedure included in the set of procedures at a position inthe input image determined based on the environment map and the positioninformation, using the acquired procedure data.

According to another embodiment of the present invention, there isprovided a program for causing a computer, which controls an imageprocessing device including a storage medium having feature data storedtherein, the feature data indicating a feature of appearance of one ormore physical objects, to function as: an environment map building unitfor building an environment map based on an input image obtained byimaging a real space using an imaging device and the feature data, theenvironment map representing a position of a physical object present inthe real space; a control unit for acquiring procedure data for a set ofprocedures of operation to be performed in the real space, the proceduredata defining a correspondence between a direction for each procedureand position information designating a position at which the directionis to be displayed; and a superimposing unit for generating an outputimage by superimposing the direction for each procedure included in theset of procedures at a position in the input image determined based onthe environment map and the position information, using the proceduredata acquired by the control unit.

As described above, according to an image processing device, an imageprocessing method, and a program in an embodiment of the presentinvention, it is possible to enable an operation procedure performed inthe real space to be intuitively and easily understood by applying theenvironment map.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic diagram showing a first example of an environmentin which an image processing device can be used according to anembodiment;

FIG. 1B is a schematic diagram showing a second example of theenvironment in which the image processing device can be used accordingto the embodiment;

FIG. 2A is a first illustrative diagram for illustrating an imageprocessing device according to an embodiment;

FIG. 2B is a second illustrative diagram for illustrating an imageprocessing device according to an embodiment;

FIG. 3A is an illustrative diagram showing an example of an input imagethat can be acquired in a real environment shown in FIG. 1A;

FIG. 3B is an illustrative diagram showing an example of an input imagethat can be acquired in a real environment shown in FIG. 1B;

FIG. 4 is a block diagram showing an example of a configuration of animage processing device according to an embodiment;

FIG. 5 is a flowchart showing an example of flow of a self-positiondetection process according to an embodiment;

FIG. 6 is an illustrative diagram for illustrating a feature point seton an object;

FIG. 7 is an illustrative diagram for illustrating addition of featurepoints;

FIG. 8 is an illustrative diagram for illustrating an example of aprediction model;

FIG. 9 is an illustrative diagram for illustrating an example of aconfiguration of feature data;

FIG. 10 is a flowchart showing an example of flow of an objectrecognition process according to an embodiment;

FIG. 11 an illustrative diagram for illustrating an example of proceduredata that can be acquired in an embodiment;

FIG. 12 is a flowchart showing an example of flow of a procedure controlprocess according to an embodiment;

FIG. 13 is a flowchart showing an example of flow of a proceduredisplaying process according to an embodiment;

FIG. 14 is an illustrative diagram showing a first example of an outputimage output by an image processing device according to an embodiment;

FIG. 15 is an illustrative diagram showing a second example of an outputimage output by an image processing device according to an embodiment;and

FIG. 16 is a block diagram showing an example of a hardwareconfiguration of a general-purpose computer.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

Also, the “detailed description of the embodiment(s)” will be describedin the following order.

1. Overview of Image Processing Device

2. Configuration of Image Processing Device according to Embodiment

-   -   2-1. Imaging unit    -   2-2. Environment Map Generating Unit    -   2-3. Output Image Generating Unit    -   2-4. Example of Output Image

3. Example of Hardware Configuration

4. Conclusion

1. OVERVIEW OF IMAGE PROCESSING DEVICE

First, an overview of an image processing device according to anembodiment of the present invention will be described with reference toFIGS. 1A to 3B. FIGS. 1A and 1B show environments 1 a and 1 b, forexample, in which the image processing device 100 according to anembodiment of the present invention can be used, respectively.

Referring to FIG. 1A, a sink 11, a cutting board 12, a bowl 13, a stove14, a microwave oven 15, and other physical objects are present in theenvironment 1 a. That is, the environment 1 a is an environmentcorresponding to a kitchen as a real space where a user prepares food.Meanwhile, referring to FIG. 1B, a table 16, chairs 17, dishes 18,glasses 19 and 20, and other physical objects are present in anenvironment 1 b. That is, the environment 1 b is an environmentcorresponding to a dining room (or guest seating of a restaurant) as areal space where a user dines.

FIGS. 2A and 2B are illustrative diagrams for illustrating an imageprocessing device 100 that may be used inside the environment 1 a or 1b, for example, as described above.

Referring to FIG. 2A, an image processing device 100 a including a body101 a, an imaging device 102 a and a display device 104 a, which aremounted to a user, is shown. The body 101 a includes a centralprocessing unit (CPU) for executing image processing in the imageprocessing device 100, which will be described in detail later, astorage medium, and the like. The imaging device 102 a is mounted to ahead portion of the user in the same direction as a user's gaze, toimage the inside of the environment 1 a or 1 b to generate a set ofinput images. The image processing device 100 a executes imageprocessing using the set of input images generated by the imaging device102 a as an input to generate a set of output images. The output imagesgenerated by the image processing device 100 a are displayed by thedisplay device 104 a. The display device 104 a is a head mount displaymounted to the head portion of the user. The display device 104 a maybe, for example, a see-through display.

Referring to FIG. 2B, an image processing device 100 b such as a mobilephone terminal held by the user is shown. The image processing device100 b is, for example, a mobile phone terminal with an imaging device,and includes an imaging device (e.g., provided at the rear of a displaydevice 104 b) and the display device 104 b. The image processing device100 b includes a CPU for executing image processing in the imageprocessing device 100, which will be described in detail later, astorage medium, and the like.

In this disclosure, when the image processing devices 100 a and 100 bneed not be discriminated from each other, letters of the referencenumerals are omitted to collectively refer to the image processingdevices 100 a and 100 b as the image processing device 100. Also, thesame applies to the imaging devices 102 a and 102 b (imaging device102), the display devices 104 a and 104 b (display device 104) and otherelements.

FIGS. 3A and 3B are illustrative diagrams showing input images 106 a and106 b, for example, acquired by the imaging device 102 imaging theenvironments 1 a and 1 b, respectively.

Referring to FIG. 3A, a cutting board 12, a stove 14, a kitchen knife21, a cruet 22 and other physical objects are shown in the input image106 a. The input image 106 a is the same image as an image shown in thevision of the user preparing food in the environment 1 a. The imageprocessing device 100 acquires a set of input images including such aninput image 106 a, for example, using the imaging device 102, andsuperimposes a direction for a cooking procedure on each input image.

Referring to FIG. 3B, a table 16, a dish 18, glasses 19 and 20, a knife23, forks 24, a napkin 25 and other physical objects are shown in aninput image 106 b. The input image 106 b is the same image as an imageshown in the vision of a user dining in the environment 1 b. The imageprocessing device 100 acquires a set of input images including such aninput image 106 b, for example, using the imaging device 102 andsuperimposes a direction for table manners during the meal on each inputimage.

An example of a configuration of such an image processing device 100will be described in greater detail in the next section.

2. CONFIGURATION OF IMAGE PROCESSING DEVICE ACCORDING TO EMBODIMENT

FIG. 4 is a block diagram showing an example of a configuration of theimage processing device 100 according to an embodiment of the presentinvention. Referring to FIG. 4, the image processing device 100 includesan imaging unit 102, an environment map generating unit 110, and anoutput image generating unit 180.

[2-1. Imaging Unit]

The imaging unit 102 may be realized as an imaging device having animaging element such as a charge coupled device (CCD) or a complementarymetal oxide semiconductor (CMOS), for example. The imaging unit 102 maybe provided outside the image processing device 100. The imaging unit102 outputs an image acquired by imaging the real space such as theenvironment 1 a or the environment 1 b to the environment map generatingunit 110 and the output image generating unit 180 as the input image.

[2-2. Environment Map Generating Unit]

The environment map generating unit 110 generates an environment maprepresenting, for example, positions of one or more physical objectspresent in the real space based on the input image input from theimaging unit 102 and feature data of an object, which will be describedlater, stored in a data storage unit 130. As shown in FIG. 4, in thisembodiment, the environment map generating unit 110 includes aself-position detecting unit 120, the data storage unit 130, an imagerecognizing unit 140, an environment map building unit 150 and anenvironment map storage unit 152.

(1) Self-Position Detection Unit

The self-position detecting unit 120 dynamically detects a position ofthe imaging device, which takes the input image, based on the inputimage input from the imaging unit 102 and the feature data stored in thedata storage unit 130. For example, even in a case in which the imagingdevice has a monocular camera, the self-position detecting unit 120 maydynamically determine a position and posture of the camera and aposition of a feature point (FP) on an imaging plane of the camera foreach frame by applying the SLAM technology disclosed in Andrew J.Davison's “Real-Time Simultaneous Localization and Mapping with a SingleCamera.” Proceedings of the 9th IEEE International Conference onComputer Vision Volume 2, 2003, pp. 1403-1410.

First, entire flow of a self-position detection process in theself-position detecting unit 120 to which the SLAM technology is appliedwill be described with reference to FIG. 5. Next, the self-positiondetection process will be described in detail with reference to FIGS. 6to 8.

FIG. 5 is a flowchart showing an example of the flow of theself-position detection process in the self-position detecting unit 120to which the SLAM technology is applied. In FIG. 5, when theself-position detection process starts, the self-position detecting unit120 first initializes a state variable (step S102). In this embodiment,the state variable is a vector including the position and the posture(rotation angle) of the camera, a moving speed and an angular speed ofthe camera and the position of one or more FPs as elements. Theself-position detecting unit 120 then sequentially obtains the inputimage from the imaging unit 102 (step S112). The process from step S112to step S118 may be repeated for each input image (i.e., each frame).

In step S114, the self-position detecting unit 120 tracks FPs present inthe input image. For example, the self-position detecting unit 120detects a patch (small image of 3×3=9 pixels around a FP, for example)of each FP stored in advance in the data storage unit 130 from the inputimage. The position of the patch detected herein, that is, the positionof the FP, is used to update the state variable later.

In step S116, the self-position detecting unit 120 generates, forexample, a predicted value of the state variable of a next frame basedon a given prediction model. Also, in step S118, the self-positiondetecting unit 120 updates the state variable using the predicted valueof the state variable generated in step S116 and an observed valueaccording to the position of the FP detected in step S114. Theself-position detecting unit 120 executes the process in steps S116 andS118 based on a principle of an extended Kalman filter.

As a result of such a process, a value of the state variable updated foreach frame is output. Hereinafter, contents of respective processes oftracking the FP (step S114), prediction of the state variable (stepS116) and updating the state variable (step S118) will be described morespecifically.

(1-1) Tracking of FP

In this embodiment, the data storage unit 130 stores the feature dataindicating features of objects corresponding to physical objects whichmay be present in the real space, in advance. The feature data includessmall images, that is, the patches regarding one or more FPs, eachrepresenting the feature of appearance of each object, for example. Thepatch may be the small image composed of 3×3=9 pixels around the FP, forexample.

FIG. 6 shows two examples of the objects and an example of FPs andpatches set on each object. A left object in FIG. 6 is the objectrepresenting a drawer (see FIG. 6a ). A plurality of FPs including afeature point FP1 are set on the object. Further, a patch Pth1 isdefined to be associated with the feature point FP1. On the other hand,a right object in FIG. 6 is the object representing a calendar (see FIG.6b ). A plurality of FPs including a feature point FP2 are set on theobject. Further, a patch Pth2 is defined to be associated with thefeature point FP2.

When the input image is acquired from the imaging unit 102, theself-position detecting unit 120 matches partial images included in theinput image against the patch for each FP illustrated in FIG. 6 storedin advance in the data storage unit 130. The self-position detectingunit 120 then specifies a position of each FP included in the inputimage (a position of a center pixel of the detected patch, for example)as the result of matching.

Further, in tracking of the FPs (step S114 in FIG. 5), data regardingall the FPs to be tracked may not be stored in the data storage unit 130in advance. For example, six FPs are detected in the input image at timeT=t−1 in an example of FIG. 7 (see FIG. 7a ). Next, when the position orthe posture of the camera changes at time T=t, only two of the six FPspresent in the input image at the time T=t−1 are present in the inputimage. In this case, the self-position detecting unit 120 may newly setFPs in positions where a characteristic pixel pattern of the input imageis present and use the new FPs in the self-position detection processfor a subsequent frame. For example, in the example shown in FIG. 7,four new FPs are set on the object at the time T=t (see FIG. 7b ). Thisis a characteristic of the SLAM technology, and accordingly, a cost ofsetting all of the FPs in advance can be reduced and accuracy of theprocess can be improved using a number of added FPs.

(1-2) Prediction of State Variable

In this embodiment, the self-position detecting unit 120 uses a statevariable X shown in the following equation as the state variable to beapplied with the extended Kalman filter.

[Equation  1] $\begin{matrix}{X = \begin{pmatrix}x \\\omega \\\overset{.}{x} \\\overset{.}{\omega} \\p_{1} \\\vdots \\p_{N}\end{pmatrix}} & (1)\end{matrix}$

The first element of the state variable X in Equation (1) represents athree-dimensional position of the camera in a global coordinate system(x, y, z) being a coordinate system set in the real space, as in thefollowing equation.

[Equation  2] $\begin{matrix}{x = \begin{pmatrix}x_{c} \\y_{c} \\z_{c}\end{pmatrix}} & (2)\end{matrix}$

Also, the second element of the state variable is a four-dimensionalvector ω having a quaternion as an element corresponding to a rotationmatrix representing the posture of the camera. The posture of the cameramay be represented using an Euler angle in place of the quaternion.Also, the third and the fourth elements of the state variable representthe moving speed and the angular speed of the camera, respectively.

Further, the fifth and subsequent elements of the state variablerepresent a three-dimensional position p_(i) of a FP FP_(i) (i=1 . . .N) in the global coordinate system as shown in a following equation.Further, as described above, the number N of the FPs may change duringthe process.

[Equation  3] $\begin{matrix}{p_{i} = \begin{pmatrix}x_{i} \\y_{i} \\z_{i}\end{pmatrix}} & (3)\end{matrix}$

The self-position detecting unit 120 generates the predicted value ofthe state variable for a latest frame based on the value of the statevariable X initialized in step S102 or the value of the state variable Xupdated in a previous frame. The predicted value of the state variableis generated according to a state equation of the extended Kalman filteraccording to multi-dimensional normal distribution shown in thefollowing equation.

[Equation 4]

predicted state variable {circumflex over (X)}=F(X,a)+w  (4)

Here, F denotes the prediction model regarding state transition of asystem, a denotes a prediction condition. Also, w denotes Gaussian noiseand may include a model approximation error, an observation error andthe like, for example. In general, an average of the Gaussian noise w is0.

FIG. 8 is an illustrative diagram for illustrating an example of theprediction model according to this embodiment. Referring to FIG. 8, twoprediction conditions in the prediction model according to thisembodiment are shown. First, as a first condition, it is assumed thatthe three-dimensional position of the FP in the global coordinate systemdoes not change. That is, provided that the three-dimensional positionof the FP FP1 at the time T is p_(T), the following relationship issatisfied.

[Equation 5]

p _(t) =p _(t−1)  (5)

Next, as a second condition, it is assumed that motion of the camera isuniform motion. That is, a following relationship is satisfied for thespeed and the angular speed of the camera from the time T=t−1 to thetime T=t.

[Equation 6]

{dot over (x)} _(t) ={dot over (x)} _(t−1)  (6)

{dot over (ω)}_(t)={dot over (ω)}_(t−1)  (7)

The self-position detecting unit 120 generates the predicted value ofthe state variable for the latest frame based on such a prediction modeland the state equation expressed in Equation (4).

(1-3) Updating of State Variable

The self-position detecting unit 120 then evaluates an error betweenobservation information predicted from the predicted value of the statevariable and actual observation information obtained as a result of FPtracking, using an observation equation, for example. Note that v inEquation (8) is the error.

[Equation 7]

observation information s=H({circumflex over (X)})+v  (8)

predicted observation information ŝ=H({circumflex over (X)})  (9)

Here, H represents an observation model. For example, a position of theFP FP_(i) on the imaging plane (u-v plane) is defined as expressed inthe following equation.

[Equation  8] $\begin{matrix}{{{position}\mspace{14mu}{of}\mspace{14mu}{FP}_{i}\mspace{14mu}{on}\mspace{14mu}{imaging}\mspace{14mu}{plane}\mspace{14mu}{\overset{\sim}{p}}_{i}} = \begin{pmatrix}u_{i} \\v_{i} \\1\end{pmatrix}} & (10)\end{matrix}$

Here, all of the position of the camera x, the posture of the camera ωand the three-dimensional position p_(i) of the FP FP_(i) are given asthe elements of the state variable X. Then, the position of the FPFP_(i) on the imaging plane is derived using the following equationaccording to a pinhole model.

[Equation 9]

λ{tilde over (p)} _(i) =AR _(m)(p _(i) −x)  (11)

Herein, λ represents a parameter for normalization, A represents acamera internal parameter, R_(ω) represents the rotation matrixcorresponding to the quaternion ω representing the posture of the cameraincluded in the state variable X. The camera internal parameter A isgiven in advance as expressed in the following equation according tocharacteristics of the imaging device, which takes the input image.

[Equation  10] $\begin{matrix}{A = \begin{pmatrix}{{- f} \cdot k_{u}} & {{f \cdot k_{u} \cdot \cot}\mspace{14mu}\theta} & u_{o} \\0 & {- \frac{f \cdot k_{v}}{\sin\mspace{14mu}\theta}} & v_{o} \\0 & 0 & 1\end{pmatrix}} & (12)\end{matrix}$

Herein, f represents focal distance, θ represents orthogonality of animage axis (ideal value is 90 degrees), k_(u) represents a scale along avertical axis of the imaging plane (rate of change of scale from theglobal coordinate system to the coordinate system of the imaging plane),k_(v) represents a scale along a horizontal axis of the imaging plane,and (u_(o), v_(o)) represents a center position of the imaging plane.

Therefore, a feasible latest state variable X may be obtained bysearching the state variable X, which makes the error between thepredicted observation information derived using Equation (11), that is,the position of each FP on the imaging plane and the result of FPtracking in step S114 in FIG. 5, minimum.

[Equation 11]

lastest state variable X←{circumflex over (X)}+Innov(s−ŝ)  (13)

The self-position detecting unit 120 outputs the position x and theposture ω of the camera (imaging device) dynamically updated by applyingthe SLAM technology in this manner to the environment map building unit150 and the output image generating unit 180.

(2) Data Storage Unit

The data storage unit 130 stores in advance the feature data indicatingthe feature of the object corresponding to the physical object, whichmay be present in the real space, using a storage medium such as a harddisk or a semiconductor memory. Although an example in which the datastorage unit 130 is a part of the environment map generating unit 110 isshown in FIG. 4, the present invention is not limited to such anexample, and the data storage unit 130 may be provided outside theenvironment map generating unit 110. FIG. 9 is an illustrative diagramfor illustrating an example of a configuration of the feature data.

Referring to FIG. 9, feature data FD1 is shown as an example for theobject Obj1. The feature data FD1 includes an object name FD11, imagedata FD12 taken from six directions, patch data FD13, three-dimensionalshape data FD14 and ontology data FD15.

The object name FD11 is the name by which a corresponding object may bespecified such as a “coffee cup A.”

The image data FD12 includes six image data obtained by taking images ofthe corresponding object from six directions: front, back, left, right,above and below, for example. The patch data FD13 is a set of smallimages around each FP for each of one or more FPs set on each object.The image data FD12 and the patch data FD13 may be used for an objectrecognition process in the image recognizing unit 140, which will bedescribed later. Also, the patch data FD13 may be used for theabove-described self-position detection process in the self-positiondetecting unit 120.

The three-dimensional shape data FD14 includes polygon information forrecognizing a shape of the corresponding object and three-dimensionalpositional information of FPs. The three-dimensional shape data FD14 maybe used for an environment map building process in the environment mapbuilding unit 150, which will be described later.

The ontology data FD15 is the data that may be used to support theenvironment map building process in the environment map building unit150, for example. In the example of FIG. 9, the ontology data FD15indicates that the object Obj1, which is the coffee cup, is more likelyto come in contact with an object corresponding to a desk or adishwasher and is less likely to come in contact with an objectcorresponding to a bookshelf.

(3) Image Recognizing Unit

The image recognizing unit 140 specifies objects to which physicalobjects present in the input image correspond, using the above-describedfeature data stored in the data storage unit 130.

FIG. 10 is a flowchart showing an example of flow of the objectrecognition process in the image recognizing unit 140. Referring to FIG.10, first, the image recognizing unit 140 acquires the input image fromthe imaging unit 102 (step S212). Next, the image recognizing unit 140matches partial images included in the input image against patches ofone or more FPs of each object included in the feature data to extractFPs included in the input image (step S214). The FPs used in the objectrecognition process in the image recognizing unit 140 and the FPs usedin the self-position detection process in the self-position detectingunit 120 are not necessarily the same. However, when common FPs are usedin the both processes, the image recognizing unit 140 may reuse theresult of FP tracking by the self-position detecting unit 120.

Next, the image recognizing unit 140 specifies the object present in theinput image based on the result of extracting the FP (step S216). Forexample, when the FPs belonging to one object are extracted with highdensity in a certain area, the image recognizing unit 140 may recognizethat the object is present in the area. The image recognizing unit 140outputs the object name (or an identifier) of the specified object andthe position of the FP belonging to the object on the imaging plane tothe environment map building unit 150 (step S218).

(4) Environment Map Building Unit

The environment map building unit 150 generates the environment mapusing the position and the posture of the camera input from theself-position detecting unit 120, the positions of the FPs on theimaging plane input from the image recognizing unit 140 and the featuredata stored in the data storage unit 130. In this disclosure, theenvironment map is a set of data indicating positions (and postures) ofone or more objects present in the real space. The environment map mayinclude object names corresponding to objects, the three-dimensionalpositions of FPs belonging to objects and the polygon informationconfiguring shapes of objects, for example. The environment map may bebuilt by obtaining the three-dimensional position of each FP accordingto the above-described pinhole model from the position of the FP on theimaging plane input from the image recognizing unit 140, for example.

By changing the relation equation of the pinhole model expressed inEquation (11), the three-dimensional position pi of the FP FP_(i) in theglobal coordinate system may be obtained by the following equation.

[Equation  12] $\begin{matrix}{p_{i} = {{x + {\lambda \cdot R_{\omega}^{T} \cdot A^{- 1} \cdot {\overset{\sim}{p}}_{i}}} = {x + {{d \cdot R_{\omega}^{T}}\frac{A^{- 1} \cdot {\overset{\sim}{p}}_{i}}{{A^{- 1} \cdot {\overset{\sim}{p}}_{i}}}}}}} & (14)\end{matrix}$

Herein, d denotes a distance between the camera and each FP in theglobal coordinate system. The environment map building unit 150 maycalculate such a distance d based on the positions of at least four FPson the imaging plane and the distance between the FPs for each object.The distance between the FPs is stored in advance in the data storageunit 130 as the three-dimensional shape data FD14 included in thefeature data described with reference to FIG. 9. It should be noted thata calculation process of the distance d in Equation (14) is disclosed indetail in Japanese Patent Application Laid-Open Publication No.2008-304268

After the distance d is calculated, remaining variables of a right sideof Equation (14) are the position and the posture of the camera inputfrom the self-position detecting unit 120 and the position of the FP onthe imaging plane input from the image recognizing unit 140, all ofwhich are known. The environment map building unit 150 then calculatesthe three-dimensional position in the global coordinate system for eachFP input from the image recognizing unit 140 according to Equation (14).The environment map building unit 150 then builds a latest environmentmap according to the three-dimensional position of each calculated FPand allows the environment map storage unit 152 to store the builtenvironment map. It should be noted that, at that time, the environmentmap building unit 150 may improve accuracy of the data of theenvironment map using the ontology data FD15 included in the featuredata described with reference to FIG. 9.

The environment map storage unit 152 stores the environment map built bythe environment map building unit 150 using the storage medium such asthe hard disk or the semiconductor memory.

[2-3. Output Image Generating Unit]

The output image generating unit 180 generates, for a set of proceduresof the operation to be performed in the real space, an output image forpresenting the direction for each procedure to the user and displays thegenerated output image on the display device 104. In this case, theoutput image generating unit 180 associates a given position in theenvironment map generated by the environment map generating unit 110with the direction for each procedure, and superimposes the directionfor each procedure at a position in the input image corresponding to thegiven position. As shown in FIG. 4, in the present embodiment, theoutput image generating unit 180 includes a procedure storage unit 184,a procedure control unit 186, an information arrangement unit 190, and asuperimposing unit 192.

(1) Procedure Storage Unit The procedure storage unit 184 storesprocedure data defined by causing the direction for each procedure tocorrespond to the position information designating a position at whichthe direction is to be displayed, for a set of procedures of operationto be performed in the real space, using a storage medium such as harddisk or a semiconductor memory.

FIG. 11 is an illustrative diagram for illustrating procedure data 182,for example, stored in the procedure storage unit 184 according to thepresent embodiment. Referring to FIG. 11, the procedure data 182 hasfive data items such as “name of directions.” “procedure ID,” “proceduredirection,” “related physical object” and “progress condition.”

The “name of directions” is a name assigned to each operation to beperformed in the real space. Procedure data about a set of proceduresfor one operation can be acquired by designating one name of directions.In the example of FIG. 11, procedure data of two operations: “tablemanners” and “frying eggs” is included in the procedure data 182.

The “procedure ID” is an identifier for identifying each procedureincluded in the set of procedures. In the example of FIG. 11, theprocedure data for the “table manners” includes three procedures ofprocedure IDs: “P101,” “P102” and “P103,” Also, the procedure data for“frying eggs” includes four procedures of procedure IDs: “P201,” “P202,”“P203” and “P204.”

The “procedure direction” is a character string indicating the directionfor each procedure to be present to the user. In the example of FIG. 11,the procedure direction of the procedure of procedure ID=“P101” of the“table manners” (hereinafter, referred to as procedure P101) is acharacter string: “sit from a left side of a chair.” The proceduredirection of procedure P102 is a character string: “order food anddrink.” The procedure direction of procedure P103 is a character string:“set a napkin on your lap.” Also, the procedure direction of procedureP201 of “frying eggs” is a character string: “divide and stir eggs.” Theprocedure direction of the procedure P202 is a character string: “movethe frying pan to the stove.” The procedure direction of the procedureP203 is a character string: “put the eggs onto the frying pan.” Theprocedure direction of procedure P204 is a character string: “sprinklesalt and pepper.”

The “related physical object” is position information designating aposition at which each procedure direction is to be displayed, using anobject name for specifying a physical object associated with eachprocedure direction. For example, in the present embodiment, eachprocedure direction is displayed in the vicinity of the physical objectspecified by the “related physical object.” In the example of FIG. 11,the related physical object of procedure P101 of the “table manners” isa “chair.” That is, the procedure direction of procedure P101 can bedisplayed in the vicinity of the chair included in the environment map.The related physical object of procedure P102 is a “menu.” That is, theprocedure direction of the procedure P102 can be displayed in thevicinity of the menu included in the environment map. The relatedphysical object of the procedure P103 is a “napkin.” That is, theprocedure direction of the procedure P103 can be displayed in thevicinity of the napkin included in the environment map. Further, forexample, the “related physical object” for a procedure not associatedwith any physical object is blank and the procedure direction of thisprocedure may be displayed in a specific position such as a center ofthe screen.

The “progress condition” indicates a condition for progressing adisplaying process from each procedure to a next procedure when theimage processing device 100 displays a set of procedures. That is, theprogress condition may be said to be information designating timing wheneach procedure direction is to be displayed. In the example of FIG. 11,the progress condition is defined according any one or combination ofthe following three patterns.

-   -   First pattern: a state of the physical object in the environment        map    -   Second pattern: passage of a given time    -   Third pattern: an external event

The first pattern is a pattern in which the progress condition issatisfied (i.e., the displaying process proceeds to a next procedure)when the position or posture of the physical object represented by theenvironment map is in a given state. For example, the progress conditionof the procedure P102 of the procedure data 182 is that “a menu is putdown.” Such a progress condition may be satisfied, for example, when themenu is not present on the table in the environment map. Also, forexample, the progress condition of the procedure P103 of the proceduredata 182 is that “the napkin is on the chair.” Such a progress conditionmay be satisfied, for example, when in the environment map when thenapkin moves from on the table to over the chair (e.g., on the lap ofthe user seated on the chair).

The second pattern is a pattern in which the progress condition issatisfied when a given time has elapsed from displaying of a precedingprocedure direction or generation of another event. For example, theprogress condition of the procedure P201 of the procedure data 182 is a“passage of one minute.” Such a progress condition may be satisfied whenone minute has elapsed after the procedure direction of the procedureP201 is displayed. Also, for example, the progress condition of theprocedure P203 of the procedure data 182 is a “passage of 30 secondsfrom stove ignition.” Such a progress condition may be satisfied, forexample, when 30 seconds have elapsed from a stove ignition eventnotified of from an external device (stove).

The third pattern is a pattern in which the progress condition issatisfied when there has been a given event notification from anotherdevice. For example, the progress condition of the procedure P202 of theprocedure data 182 is that “the frying pan is on the stove.” Such aprogress condition may be satisfied, for example, when the event isnotified of from the stove having detected the frying pan being on thestove, for example, using a pressure sensor. Further, the satisfactionof the condition: “there is a frying pan on a stove” may be determinedfrom a state of the physical object in the environment map (in thiscase, which is the first pattern). An event notification from anotherdevice is not limited to such an example, but may be, for example, anotification of the image recognition result (e.g., the result ofrecognizing a state of a user's hand) from the imaging device, anotification about a user manipulation from an electronic device, anotification about signal reception from a communication device, or thelike.

The procedure storage unit 184 stores such procedure data 182 as anexample, and outputs at least part of the procedure data 182 accordingto a request from the procedure control unit 186, which will bedescribed later. Further, in the example of FIG. 4, the procedurestorage unit 184 is provided inside the image processing device 100.However, the prevent invention is not limited to such an example, butthe procedure data may be stored in a storage medium external to theimage processing device 100. In this case, the image processing device100 may selectively acquire necessary procedure data from the externalstorage medium, for example, according to an instruction of the user.Also, a plurality of procedures whose states transition from each otheras in a state machine may be defined as procedure data instead of asequential procedure such as the procedure data 182 illustrated in FIG.11.

(2) Procedure Control Unit

The procedure control unit 186 acquires procedure data for a desiredname of directions from the procedure storage unit 184, for example,according to an instruction from the user, and controls displaying ofthe procedure direction for each procedure included in the set ofprocedures according to the acquired procedure data.

FIG. 12 is a flowchart showing an example of flow of a procedure controlprocess in the procedure control unit 186 according to the presentembodiment. Referring to FIG. 12, first, the procedure control unit 186acquires procedure data for a desired name of directions from theprocedure storage unit 184 (step S302). For example, when the userdesignates “table manners” as the name of directions, the procedurecontrol unit 186 acquires procedure data for the “table manners”including the set of procedures P101, P102, P103, . . . illustrated inFIG. 11.

Next, the procedure control unit 186 reads a first procedure from theacquired procedure data (step S304). For example, when the proceduredata including the set of procedures P101, P102, P103 . . . illustratedin FIG. 11 is acquired, the procedure control unit 186 initially reads arecord of the procedure P101.

Next, the procedure control unit 186 displays a procedure direction inthe vicinity of a related physical object corresponding to the readprocedure (step S306). More specifically, the procedure control unit 186specifies, for example, in the environment map, a position of therelated physical object corresponding to the read procedure, anddetermines the vicinity of the position of the related physical objectas a three-dimensional position at which the procedure direction is tobe displayed. The procedure control unit 186 outputs the proceduredirection and the three-dimensional position at which the proceduredirection is to be displayed, to the information arrangement unit 190. Asubsequent procedure displaying process will be described in greaterdetail later.

Next, the procedure control unit 186 monitors a state of the physicalobject, passage of a given time, an external event, or the like in theenvironment map according to the progress condition corresponding to theread procedure (step S308). When the progress condition is satisfied asa result of monitoring, the process proceeds to step S310.

Next, the procedure control unit 186 determines whether a next procedurethat has not been displayed remains (step S310). Here, when the nextprocedure that has not been displayed remains, the process returns tostep S304 and the procedure displaying process is repeated for a nextprocedure. On the other hand, when the next procedure that has not beendisplayed does not remain, the procedure control process in theprocedure control unit 186 ends.

(4) Information Arrangement Unit

The information arrangement unit 190 calculates a position in the inputimage at which each procedure direction input from the procedure controlunit 186 is to be displayed, according to Equation 11 of the pinholemodel using the position and posture of the imaging device acquired fromthe environment map generating unit 110. In this case, thethree-dimensional position p_(i) of the FP FP_(i) at the right side ofEquation (11) is substituted with the three-dimensional position inputfrom the procedure control unit 186. After the information arrangementunit 190 calculates the position in the input image at which eachprocedure direction is to be displayed, the information arrangement unit190 outputs each procedure direction and the position in the input imageat which the procedure direction is to be displayed, to thesuperimposing unit 192.

(5) Superimposing Unit

The superimposing unit 192 generates an output image by superimposingeach procedure direction input from the information arrangement unit 190at the position in the input image calculated by the informationarrangement unit 190.

FIG. 13 is a flowchart showing an example of flow of a proceduredisplaying process in the procedure control unit 186, the informationarrangement unit 190 and the superimposing unit 192. Further, theprocedure displaying process shown in FIG. 13 is executed in step S306in the procedure control process shown in FIG. 12 for each individualprocedure included in a set of procedures of operation to be performedin the real space.

Referring to FIG. 13, first, the procedure control unit 186 determinesthe three-dimensional position at which the procedure direction is to bedisplayed, based on a position of a related physical object in theenvironment map (step S322). For example, when the related physicalobject for any procedure is on a chair, a position of either a surfaceor the vicinity of the chair in the environment map is determined as thethree-dimensional position at which the procedure direction for theprocedure is to be displayed.

Next, the information arrangement unit 190 calculates the position inthe input image corresponding to the three-dimensional positiondetermined in step S322 according to the pinhole model using theposition and posture of the imaging device acquired from the environmentmap generating unit 110 (step S324). Further, when the position on theimaging plane corresponding to the three-dimensional position determinedin step S322 is out of a range of the input image, a subsequent processmay be skipped.

Next, the superimposing unit 192 generates an output image bysuperimposing, for example, a text box describing the proceduredirection at the position in the input image calculated by theinformation arrangement unit 190 (step S326). The output image generatedby the superimposing unit 192 is displayed, for example, on the screenof the display device 104 of the image processing device 100.

[2-4. Example of Output Image]

FIGS. 14 and 15 show examples of an output image that can be displayedon the screen of the display device 104 in the present embodiment,respectively.

Referring to FIG. 14, an output image Im11 on which the proceduredirection for the procedure P202 illustrated in FIG. 11 is superimposedis shown. In the output image Im11, a text box T11 describing aprocedure direction “procedure 2: move the frying pan to the stove” isdisplayed in the vicinity of the frying pan. The user can intuitivelyrecognize that it is good for the frying pan to be on the stove in anext cooking procedure by viewing such a procedure direction. This textbox T11 continues to be displayed in the vicinity of the frying panuntil a state “the frying pan is on the stove,” which is the progresscondition of the procedure P202, is realized in the environment map.Thus, the user can easily recognize a physical object serving as anobject of the procedure.

Referring to FIG. 15, an output image Im21 on which a proceduredirection for the procedure P103 illustrated in FIG. 11 is superimposedis shown. In the output image Im21, a text box T21 describing aprocedure direction “procedure3: set a napkin in your lap” is displayedin the vicinity of the napkin. The user can intuitively recognize thatit is good for the napkin to be set on the lap in a next procedure ofthe table manners by viewing such a procedure direction. This text boxT21 continues to be displayed in the vicinity of the napkin until astate “the napkin is on the chair.” which is the progress condition ofthe procedure P103, is realized in the environment map. Thus, the usercan easily recognize a physical object serving as an object of theprocedure.

The procedure display method described herein is only an example, andvarious applications are possible, in addition to the above-describedembodiment. For example, the present invention may be applied fordirections in spaces such as commercial facilities or train stations orfor manipulation directions in a rent-a-car or an automated tellermachine (ATM).

3. EXAMPLE OF HARDWARE CONFIGURATION

It does not matter whether a set of processes according to theabove-described embodiment are realized by hardware or software. When aset of processes or a part of the same is executed by the software, aprogram composing the software is executed using a computer incorporatedin dedicated hardware or a general-purpose computer shown in FIG. 16,for example.

In FIG. 16, the CPU 902 controls entire operation of the general-purposecomputer. A program or data describing some or all of the processes inthe set is stored in a read only memory (ROM) 904. The program and dataused by the CPU 902 in process execution are temporarily stored in arandom access memory (RAM) 906.

The CPU 902, the ROM 904, and the RAM 906 are connected to each othervia a bus 910. Further, an input/output interface 912 is connected tothe bus 910.

The input/output interface 912 is an interface for connecting the CPU902, the ROM 904, and the RAM 906 with the input device 920, the displaydevice 104, the storage device 924, the imaging device 102, and thedrive 930.

The input device 920 receives instructions or information input from theuser, for example, via an input interface such as a button, a switch, alever, a mouse, or a keyboard. The storage device 924 includes, forexample, a hard disk drive or a semiconductor memory, and storesprograms or data. The drive 930 is provided in the general-purposecomputer as necessary and, for example, a removable media 932 is mountedin the drive 930.

When the set of processes is executed by software, for example, aprogram stored in the ROM 904, the storage device 924, or the removablemedia 932 shown in FIG. 16, when executed, is read into the RAM 906 andthen executed by the CPU 902.

4. CONCLUSION

The embodiment of the present invention has been described withreference to FIGS. 1A to 16. According to the present embodiment, thedirection for each procedure included in the set of procedures of theoperation to be performed in the real space is displayed at the positiondetermined based on the environment map, which three-dimensionallyrepresents a position of a physical object present in the real space.Accordingly, the procedure can be intuitively understood by the user. Inparticular, according to the present embodiment, the environment map isdynamically updated to follow a change in the environment even in theoperation in the real space in which an environment surrounding the usermay be dynamically changed. As a result, the position at which thedirection for each procedure is displayed moves according to the changeof the environment, thereby preventing difficulty in understanding ofthe directions from being caused due to the change of the environment.

Also, in the present embodiment, the position at which the direction foreach procedure is to be displayed can be determined according to theposition of the physical object associated with the direction for eachprocedure. Accordingly, it is possible for the user to easily recognizethe physical object as a target object of the operation in eachprocedure. Also, since the timing when the direction for each procedureis to be displayed is controlled according to the state (position orposture) of the physical object in the environment map, passage of agiven time, an external event or the like, the direction can bedisplayed at proper timing according to the progress of the operation bythe user. Further, the position at which the direction for eachprocedure is to be displayed is calculated based on the position andposture of the camera dynamically detected using SLAM technology.Accordingly, even when a camera whose position or posture is likely tobe changed from moment to moment is used, the direction can be displayedin a proper position in the image.

The preferred embodiments of the present invention have been describedabove with reference to the accompanying drawings, whilst the presentinvention is not limited to the above examples, of course. A personskilled in the art may find various alternations and modificationswithin the scope of the appended claims, and it should be understoodthat they will naturally come under the technical scope of the presentinvention.

What is claimed is:
 1. An image processing device comprising: aprocessor; and a non-transitory computer-readable medium encoded withinstructions which, when executed by the processor, cause the imageprocessing device to: acquire procedure data for a set of procedures ofoperation to be performed in a real space and which associates aprocedure of the set of procedures with an object; control, based on theacquired procedure data, a display of first procedure information bysuperimposing a first procedure of the set of procedures; detect a firstprogress condition when the first procedure information is displayed;and control, in a case that the first progress condition is detected, adisplay of second procedure information by superimposing a secondprocedure of the set of procedures, based on the acquired proceduredata, wherein the first procedure information is related to a firstobject in the real space, and wherein the second procedure informationis related to a second object in the real space.
 2. The image processingdevice according to claim 1, wherein the first progress condition isdetected based on whether or not a predetermined time has elapsed. 3.The image processing device according to claim 1, wherein the firstprogress condition is detected based on whether or not a predeterminedtime has elapsed from detecting a user operation.
 4. The imageprocessing device according to claim 1, wherein the first procedureinformation is displayed as a text box.
 5. The image processing deviceaccording to claim 1, wherein the first progress condition is detectedbased on a result of recognizing a state of a hand of a user.
 6. Theimage processing device according to claim 1, wherein the non-transitorycomputer-readable medium is further configured to control a display ofthe first procedure information at a position associated with the firstobject in the real space.
 7. The image processing device according toclaim 1, wherein the procedure data defines a correspondence between adirection for each procedure and position information designating aposition at which the direction is to be displayed or a condition forprogressing displaying of each direction.
 8. The image processing deviceaccording to claim 7, wherein the condition for progressing displayingof each direction includes a condition according to a position or aposture of the object.
 9. The image processing device according to claim7, wherein the displaying of the direction for each procedure includedin the set of procedures is controlled according to the procedure data.10. The image processing device according to claim 1, wherein the firstprogress condition comprises at least one of a state of the object,passage of a given time, an external event, or a combination thereof.11. The image processing device according to claim 1, wherein the firstprocedure information is displayed prior to the second procedureinformation being displayed.
 12. The image processing device accordingto claim 1, wherein the non-transitory computer-readable medium isfurther configured to store feature data that indicates features ofappearance of objects, and wherein the first object and the secondobject are identified based on correspondence between the stored featuredata.
 13. The image processing device according to claim 1, wherein theset of procedures comprises more than two procedures.
 14. The imageprocessing device according to claim 13, wherein the image processingdevice is further caused to: detect a second progress conditionassociated with the second procedure when the second procedureinformation is displayed; and control, based on the acquired proceduredata and when the second progress condition is detected, a display ofthird procedure image data by superimposing a third procedure of the setof procedures, wherein the first procedure information is displayedprior to the second procedure information being displayed, and thesecond procedure information is displayed prior to the third procedureimage data being displayed, and wherein the third procedure image datais related to a third object in the real space.
 15. The image processingdevice according to claim 1, wherein the set of procedures comprisestext based information.
 16. An image processing method comprising:acquiring procedure data for a set of procedures of operation to beperformed in a real space and which associates a procedure of the set ofprocedures with an object; controlling, based on the acquired proceduredata, a display of first procedure information by superimposing a firstprocedure of the set of procedures; detecting a first progress conditionwhen the first procedure information is displayed; and controlling, in acase that the first progress condition is detected, a display of secondprocedure information by superimposing a second procedure of the set ofprocedures, based on the acquired procedure data, wherein the firstprocedure information is related to a first object in the real space,and wherein the second procedure information is related to a secondobject in the real space.
 17. A non-transitory computer-readable mediumhaving embodied thereon a program, which when executed by a computercauses the computer to execute a method, the method comprising:acquiring procedure data for a set of procedures of operation to beperformed in a real space and which associates a procedure of the set ofprocedures with an object; controlling, based on the acquired proceduredata, a display of first procedure information by superimposing a firstprocedure of the set of procedures; detecting a first progress conditionwhen the first procedure information is displayed; and controlling, in acase that the first progress condition is detected, a display of secondprocedure information by superimposing a second procedure of the set ofprocedures, based on the acquired procedure data, wherein the firstprocedure information is related to a first object in the real space,and wherein the second procedure information is related to a secondobject in the real space.